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Abstract 

The purpose of the study was to meta-synthesize research findings on the effectiveness of problem posing and to 
investigate the factors that might affect the incorporation of problem posing in the teaching and learning of 
mathematics. The eligibility criteria for inclusion of literature in the meta-analysis was: published between 1989 
and 2011, reported problem posing as an intervention, employed experimental research design, and provided 
data necessary to compute effect sizes. The large positive effect sizes (Fledges’ g) showed that problem posing 
activities provide considerable benefits for: mathematics achievement, problem solving skills, levels of problems 
posed, and attitudes toward mathematics. Several noteworthy limitations of this study were discussed. 

Keywords: mathematical problem posing, teaching strategies, meta-analysis 

1. Introduction 

In recent years, mathematical problem posing has been gaining considerable attention as a useful cognitive 
activity along with problem solving. The National Council of Teachers of Mathematics (NCTM) highlighted the 
importance of problem posing as a classroom intervention strategy for reforming school mathematics. 
Mathematics educators have acknowledged problem posing as a worthy intellectual activity based on 
constructivist theories of teaching and learning (Silver, 1994). According to English (1997), when students pose 
their own problems, they can enhance their mathematical knowledge, stimulate critical thinking, and improve 
computational skills by exploring their curiosity about specific mathematics concepts. 

Problem posing is considered a developmental tool for critical thinking (English, 1997; Lowrie, 2002) because it 
can help students extend what they know in order to develop mathematical fluency and engage them in 
higher-order thinking (NCTM, 2000). In order to create an effective problem, posers must have imaginative 
skills that can be developed through the process of problem solving (Kilpatrick, 1987). Lavy and Bershadsky 
(2003) stated students not only needed to think mathematically but also think creatively when reformulating and 
generating a new mathematical problem. These researchers believed that students who were engaged in problem 
posing activities became enterprising, creative, and active learners. 

2. Problem Posing in the Teaching and Learning of Mathematics 

Research on problem posing has been shown to have positive outcomes on students’ knowledge, problem 
solving skills, problem posing abilities, creativity, and disposition toward mathematics (e.g., Cai, 1998, 2003; 
Cai & Flwang, 2002; English, 1997, 1998; Lavy & Bershadsky, 2003; Moses, Bjork, & Goldenberg, 1990; Silver, 
Mamona-Downs, Leung, & Kenney, 1996; Stoyanova, 1999; Yuan & Sriraman, 2011). For instance, Lavy and 
Bershadsky (2003) discovered that the use of problem posing activities contributed to the development of an 
individual’s mathematical knowledge. In this study, the What if not? strategy (Brown & Walter, 2005) was 
adapted into two learning workshops on complex solid geometry. The results showed that students have deeper 
understanding of geometry and strengthen some interrelated mathematical concepts which supported Moses et 
al.’s (1990) ideas of restructuring and connecting knowledge based on prior ones with the use of problem posing 
activities. 

In addition, a number of empirical works reported the effects of problem posing activities on attitudes and beliefs 
about mathematics and mathematics instruction. In their study, Barlow and Cates (2006) found that when 
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teachers incorporated a problem-posing intervention, classrooms became more student-centered and students 
were more actively involved in creating and solving their own problems. On the other hand, problem posing 
gave students ownership of the problems they had generated or formulated (Cunningham, 2004; Grundmeier, 
2003; Within, 2006). Cunningham stated that when students created new problems, they increased their sense of 
responsibility as they constructed their own knowledge while critiquing and refining problems with their 
classmates. By encouraging students to share and critique their ideas during mathematical activities, a teacher 
can develop the dispositions and inquiry skills students need to help them develop as effective problem posers 
(English, 1997; Gonzales, 1998; Silver, 1994). 

Despite the significant contributions of this cognitive activity for improving student learning in mathematics 
(NCTM, 2000), very few classroom teachers have students regularly create their mathematical problems (Crespo 
& Sinclair, 2008; Silver, 1994). In fact, students of all ages, including new teachers, have limited experience 
posing their own mathematical problems (Crespo & Sinclair, 2008). For instance, Silver et al. (1996) found that 
53 middle school teachers and 28 prospective secondary school teachers were able to generate at least one 
mathematical problem but many were “ill-posed or poorly stated problems” (p. 305). The results of this study 
suggested that even though the participants lacked educational experience, it was not a constraint for adapting 
problem posing in classrooms. However, Silver et al. (1996) strongly believed that when teachers do not have a 
certain level of proficiency to pose mathematical problems, it is difficult for them to involve and engage in this 
activity. 

Because teachers have an essential role in the implementation of problem posing in classrooms, they must 
develop skills in problem posing and be able to create tasks, which include situations that are suitable for student 
engagement and participation (Gonzales, 1996; Silver et al., 1996). In fact, the NCTM’s Principles and 
Standards for School Mathematics (2000) stated that teachers are responsible for creating a learning environment 
that promotes student discourse and provokes their critical thinking through problem posing activities. Many 
mathematics educators, therefore, have suggested the incorporation of problem posing during teacher education 
for creating more opportunities for teachers and preservice teachers to learn and experience the process that will 
eventually impact their future classroom pedagogical strategies. In fact, Stoyanova (2005) argued that problem 
posing was similar to other skills that can be developed and nurtured over time from structured to less structured 
activities. With enough exposure to problem posing activities, anyone should be able to pose their own problems 
that are mathematically rich, realistic, and challenging (Chen, Van Dooren, Chen, & Verschaffel, 2010). 

3. The Need for a Meta-Analytic Review 

A large body of literature has examined how problem-posing activities were integrated in mathematics 
classrooms, however, much remains to be understood about the types of classroom settings, type of presentation, 
and students that would most benefit from teaching and learning mathematics with problem posing. To date, 
there is only one review of the literature available providing an overview of published articles on problem posing. 
That review was conducted by Silver (1994), who summarized the nature, processes, and perspectives of 
problem posing activities. He discussed some insightful findings from research articles and provided 
recommendation for further investigation. Yet, Silver’s review was written 20 years ago, hence, a systematic 
approach for reviewing the body of empirical work is needed. The present study employed a meta-analysis 
methodology for integrating research findings on the effects of problem posing interventional strategies on 
student learning outcomes. It is the first meta-analytic review on this topic designed to help teachers and 
researchers in making decisions regarding problem posing as an effective classroom interventional tool for the 
teaching and learning of mathematics. 

4. Methods 

4.1 Purpose of the Present Study 

With a growing number of studies focusing on problem posing as a classroom intervention, it becomes important 
for us to examine the effectiveness of this approach for improving student mathematics learning. Therefore, the 
purpose of this study was to systematically synthesize empirical research on problem posing as an intervention in 
mathematics classrooms. The term intervention, as used in this study, describes any type of problem posing 
activity or program that might affect student learning of mathematics. In addition, we investigated the factors 
that might affect the incorporation of problem posing into the teaching and learning of mathematics. This 
meta-analytic study addresses the following questions: (a) Does problem posing lead to improvement in student 
learning of mathematics? (b) What factors (e.g., interventional features, methodological, student characteristics) 
moderate the effectiveness of problem posing in mathematics classrooms? 
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4.2 Inclusion Criteria and Literature Search 

In the U.S., problem posing gained considerable attention in 1989 when NCTM documented the reform vision 
for school mathematics with an endorsement incorporating a cognitive activity, problem posing with problem 
solving as a means of classroom intervention. Since that year, the research on mathematical problem posing has 
grown. For purposes of this meta-analysis, we located significant works starting in 1989 until 2011 in order to 
systematically review effects of the problem posing intervention on student learning of mathematics. 

The first step was to determine the meta-syntheses inclusion terms. The subject term keywords “problem 
posing”, “problem writing”, “problem formulation”, “problem generation” (with the Boolean operator of an 
asterisk) and “ mathematics ” (e.g., problem posing * and mathematics) were selected and used in the following 
web-based educational databases: Educational Resources Information Center [(ERIC) via EBSCOhost], 
ProQuest Dissertations and Theses (ProQuest), and PsycINFO (ProQuest). We explored the databases of returned 
citations looking for works that focused on mathematical problem posing as a classroom intervention. Additional 
studies were located by examining the reference lists of the selected studies. Also, a manual search was 
conducted of a number of leading journals in education, particularly in mathematics education such as 
Mathematical Thinking and Learning, Educational Studies in Mathematics, Journal for Research in 
Mathematics Education, Journal of Mathematical Behavior, School Science and Mathematics, For Learning of 
Mathematics, and American Educational Research Journal. Finally, we used the Advanced Google Scholar 
search program to cross-validate article saturation. 

The multiple searches yielded an initial pool of 3,730 works, however, many of these studies were duplicates, or 
not applicable to the topic under investigation, or untraceable. We scanned the title of each work and sorted the 
potential papers. This resulted in 167 articles from journals, proceedings, ERIC documents, and dissertations 
related to mathematical problem posing. The next step was to examine each article in depth, using the eligibility 
criteria. The eligibility criteria for inclusion were: (a) reported problem posing as an intervention without the aid 
of a computer system/program, (b) employed experimental research design (i.e., treatment and control groups or 
pretest-posttest group), (c) published in the English language, and (d) reported statistical data (e.g., mean, 
standard deviation, frequency, proportion, t-test, correlation coefficient) for calculation of effect sizes. Many 
articles were discarded because they adopted non-experimental research designs (n=71; e.g., Cai, 1998; Yuan & 
Sriraman, 2011), presented only theoretical frameworks and literature reviews (n= 23; Christou, Mousoulides, 
Pittalis, & Pitta-Pantazi, 2005; Contreras, 2007), and documented pedagogical approaches («=43; e.g., Cai & 
Brook, 2006). We also excluded two studies that did not report the appropriate statistical data to compute effect 
sizes (i.e., Barlow & Cates, 2006) and 13 studies that utilized a computer system/program to enhance problem 
posing (e.g., Hirashima, Yokoyama, Okamoto, & Takeuchi, 2007). After screening the abstracts and careful 
examination of each remaining work, only 14 individual studies were selected because they satisfied the 
requirements for this meta-analysis. 

4.3 Coding Procedure 

The retained studies were coded using a 35-item coding sheet, which was developed iteratively based on the 
information provided in the studies. The researchers coded the articles according to important variables that 
illustrated the characteristics of the study. For example, the independent variables coded in the studies were 
authors, year of publication, type of publication, sample size, grade level, task administration, instruments, type 
of treatment, and statistical analyses used. Some particular research studies did not provide enough information 
for coding; hence, we contacted the authors to gather data about their important variables and research design. 
The dependent variables included the type of student mathematical learning outcome (i.e., knowledge-based, 
ability-based, skill-based, attitude and belief) resulting from the problem posing intervention. We categorized the 
learning outcome as knowledge-based when a study reported an effect of problem posing on student 
achievement in mathematics content. For ability-based measure, we coded an intervention that had an impact on 
students’ levels and kinds of problem they posed. When the participants were able to perform some computation 
and apply problem-solving skills, we classified them as a skill-based measure. Finally, data collected about 
students’ attitudes and beliefs about or toward mathematics, we coded as an attitude-beliefs outcome. 
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Table 1. Studies included in the meta-analysis of problem posing 


Study 

(years) 

Source 

N 

Participants 

(Location) 

Learning Outcome (area) 

Problem Posing Intervention 

Abu-Elwan 

(2002) 

Journal of Science 

and Mathematics 

Education in S.E. 

Asia 

50 

M. Gr. PTs 
(Oman) 

SB (problem solving), AB 
(perfonnance in problem 
posing) 

What if not?, modify givens 

Akay & 

Australian Journal of 

82 

E. PTs 

Attitude (math, attitude & 

Semi-structured and 

Boz (2010) 

Teacher Education 


(Turkey) 

self-efficacy) 

structured pp. situations, 
free pp. situation 

Demir 

Maters’ thesis 

82 

Gr. 10 

KB (probability achievement), 

What if not?, 

(2005) 



(Turkey) 

Attitude (probability and 

semi-stnictured and 





math.) 

structured pp. situations, 
free pp. situation 

Dickerson 

Doctoral dissertation 

200 

Gr. 7 (U.S.) 

SB (problem solving 

What if not?, structured, 

(1999) 




perfonnance) 

acting-out, open-ended 

English 

Educational Studies 

27 

Gr. 4 

AB (levels of problem), SB 

Recognition of problem 

(1997) 

in Mathematics 


(Australia) 

(math, procedure) 

structures, creation of 

problem from given 

components, modeling of 
new problems on existing 
problem structures 

English 

Journal for Research 

54 

year 3 in E. 

AB (levels of problems) 

Creation of number 

(1998) 

in Mathematics 


school 


sentences problems, 


Education 


(Australia) 


generation of problems from 
non-goal-specific situations, 
from a stimulus picture, and 
from literature 

Fetterly 

Doctoral dissertation 

32 

E. PTs (U.S.) 

Attitude (mathematics anxiety 

Exposure to multiple 

(2010) 




& beliefs), AB (creativity) 

perspectives, pose an 
open-ended problem, 

examine sample solutions, 
and pose alternative 

problems. 

Grundmeier 

Doctoral dissertation 

19 

E. & M. Gr. 

AB (levels of problem) 

A five-step problem solving 

(2003) 



PTs (U.S.) 


heuristic (pose a related 
problem), problem 

generation and problem 
refonnulation 

Kesan, 

International Online 

64 

Gr. 8 

KB (math, performance ) 

No specific pp. strategy 

Kaya, & 

Journal of 


(Kazakhstan) 



Guvercin 

(2010) 

Educational Sciences 





Priest 

Doctoral dissertation 

31 

Year 7 

SB (problem solving 

Structured pp. 

(2009) 



(Australia) 

competence) 

activities-problem 


generation, exploration 


230 




www.ccsenet.org/ies 


International Education Studies 


Vol. 7, No. 13; 2014 


Schloemer 

Doctoral dissertation 

40 

Gr. 10, 11 

KB (advanced algebra 

What if not?, cognitive 

(1994) 



(U.S.) 

achievement), AB (pp. 

performance). Attitude (math, 
disposition) 

apprenticeship 

Toluk-Ucar 

Teaching and Teacher 

95 

Gr. 5-8 PTs 

KB (math, achievement), AB 

No specific pp. strategy 

(2009) 

Education 


(Turkey) 

(levels of problem), SB 
(representation, math, 

procedures). Attitude (view 
about math, knowledge) 


Xia, Lu, & 

Journal of 

327 

Junior high 

Attitude (interest in math.) 

No specific pp. strategy 

Wang 

Mathematics 


school 



(2008) 

Education 
(Experiment 1) 


(China) 



Xia, Lu, & 

Journal of 

213 

Gr. 9 (China) 

AB (levels of problem) 

No specific pp. strategy 

Wang 

Mathematics 





(2008) 

Education 
(Experiment 2) 






Note. Gr. = Grade, E. = Elementary, M. = Middle, PTs = Preservice Teachers, math. = mathematics, pp. = 
problem posing, AB = ability-based, KB = knowledge-based, SB = skill-based. Attitude = Attitude-beliefs. 


In this meta-analytic study, two researchers reviewed and coded two articles for assessing inter-rater reliability. 
The inter-rater coding was 80% and discussions were conducted to resolve disagreements related to coding 
procedure resulting in a 100% agreement. Then, one of these researchers coded the remaining articles. The 
description of the selected articles is illustrated in Table 1. After the coding process, we computed an effect size 
index for each learning outcome in a study. According to Lipsey and Wilson (2001), the effect size was an 
essential feature of meta-analysis in order to statistically combine and compare the studies that use different 
measurement procedures. The calculation was based on the standardization concept of different statistical results 
in each of the previous research studies. This standardized score (effect size) was interpretable and meaningful 
when making assessments across the variety of studies. 

4.4 Computing Effect Sizes 

The obtained effect sizes were based on seven published articles, five doctoral dissertations, and a master’s thesis 
included in this meta-analysis. These studies employed an experimental research design for two independent 
groups or matched groups (pretest-posttest) and reported frequencies, proportions, and/or means for illustrating 
“two variable relationships” (Lipsey & Wilson, 2001, p. 41). 


Table 2. Description of data, research designs, and effect sizes 


Data 

Research Design 

Effect Size Indices 

Continuous 

Independent groups 

Standardized mean difference 

Continuous 

Matched groups 

Standardized mean gain 

Dichotomous 

Independent groups 

Odds ratio 


In order to accommodate the variability of study designs and data types reported in each study, three effect size 
indices were included in this meta-analytic study as presented in Table 2. Borenstein, Hedges, Higgins, and 
Rothstein (2009) stated, “A single systematic review can include studies that used independent groups and also 
studies that used matched groups. From a statistical perspective the effect size has the same meaning regardless 
of the study design” (p. 30). A transformation for adjusting the indices used for different types of data is needed 
before combining these estimates in a meta-analytic study (Borenstein et al., 2009; Lipsey & Wilson, 2001). 

According to Lipsey and Wilson (2001), the standardized mean difference is appropriate when contrasting 
experimental and control group means by adjusting the standard deviation of these groups to make them 


231 




www.ccsenet.org/ies 


International Education Studies 


Vol. 7, No. 13; 2014 


comparable even if the studies were using different measurement. In addition, the computation of standardized 
mean difference effect sizes from summary statistics (e.g., t-statistic, chi-square) is preferable to account for 
covariates. Then, the standardized mean gain is used when contrasting the matched group means that utilized the 
pooled standard deviation and correlation between the pre and posttest (Lipsey & Wilson, 2001). The odds ratio 
formula is an effect size metric for independent group research studies that uses dichotomous variables for which 
the data were presented in terms of frequencies and proportions (Lipsey & Wilson, 2001). Two studies that 
utilized a matched group research design (pre-posttest only) but reported only proportions and frequencies as 
their results were excluded. These were excluded because we found it impossible to extract the data into a 2 x 2 
table for finding an effect size index, dependent risk difference or dependent proportions difference 
(Sanchez-Meca, personal communication, November 27, 2011). In addition, we were unable to find any citations 
that offered the possibility of converting and comparing dependent risk difference into Hedges’ g or odds ratio 
indices. 

4.5 Statistical Analysis 

With careful examination of the variation of data in each article, a table was developed in Microsoft Office Excel 
(Ms-Excel) version 2007 to organize the data for ease of computation. The calculation of effect size was 
performed on an individual basis because some of the studies employed more than one outcome measure of 
problem posing effectiveness within a study. First, we calculated the Glass’ A effect size and standard error for 
each learning outcome in a study. The weight function was used to weight the effect sizes by sample sizes 
(Lipsey & Wilson, 2001). If the sample sizes were equal, the mean effect was the average of the effects (Lipsey 
& Wilson, 2001). Next, we applied the Hedges’ g adjusted estimate to each effect size index for correcting the 
sampling bias which gave more weight to studies with larger samples (Lipsey & Wilson, 2001). If the study 
examined a key outcome construct in several ways, an average of the overall effect sizes was computed. Then, 
we adapted Hasselblad and Hedges’ (1995) ideas of converting among effect sizes because some indices were on 
different scales, continuous and dichotomous (Lipsey & Wilson, 2001). For ease of interpretation of results and 

TCE n 
(-—) 

discussion, we converted all odds ratio to Hedges’ g indices based on E or = e V5 where E or and E g are odds 
ratio and Hedges’ g effect sizes, respectively. 

In the final stage of data analysis, the Hedges’ g indices were grouped into similar research outcomes for testing 
the homogeneity analysis (Q statistics) of the effect size distribution. The test provides information about the 
variations of the true effect sizes that might have occurred because of sampling error or other aspects (Borenstein 
et al., 2009). In addition, the data were imported into Statistical Package for Social Science (SPSS) version 17.0 
(2008) and funnel plots were constructed to display the effect size distribution visually in order to detect 
publication bias as a potential threat to validity. According to Allen, Chen, Willson, and Hughes, (2009) 
publication bias happens when publication favors studies with statistically significant results over studies with 
non-statistically significant results. The funnel plot was based on the mean effect for each study against the total 
sample size, and the confidence intervals (Allen et al., 2009). Asymmetrical plots might indicate the possible 
presence of publication or other forms of selection biases; however, other factors such as the design, participants’ 
characteristics, type of treatment, and true heterogeneity may affect the shape of the funnel plot as well (Tang & 
Liu, 2000). 

4.6 Moderator Variables 

We hypothesized four variables as potential moderators that could influence the size of effect from the problem 
posing intervention (a) educational settings (grade 3-12 versus teacher preparation); (b) research designs (group 
versus pre-post contrasts); (c) publication sources published versus unpublished); (d) presentation types 
(structured versus non-specific problem posing program). The moderator analysis was performed in Ms-Excel by 
comparing the average effect sizes of each key outcome measure in different categories that formed the levels of 
the moderator. Specifically, in this moderator analysis, we applied a fixed-effect model where in a Q test based 
on analysis of variance (ANOVA) was utilized for comparing two or more groups (Borenstein et al., 2009). For 
example, when comparing two groups (Group A with B), we computed the weighted sum of squared deviations 
of studies in each group (Q A , Q B ) and the sum of O , and Q B (Qwithin )• Then, we calculated the weighted sum of 
squared deviations of the subgroup means about the grand mean (Qbetween) and the weighted sum of squared 
deviations of all effects about the grand mean, Q. We ran moderator analyses for all learning outcomes and based 
on the calculated quantities, we hoped to determine whether the effect of a problem posing intervention differed 
in the two groups and moderators were identified in every learning outcome. 
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5. Results 

5.1 Description of Selected Studies 

A total of 14 unique datasets (from 13 studies) were included in the current meta-analysis. Table 1 provided 
study details. Five studies were published between 2009-2010, two from Turkey, and one each from the U.S., 
Kazakhstan, and Australia. The remainder were published between 1994 and 2008 from diverse countries such as 
Australia (n= 2), the U.S. (/?=3), and one each from Oman and China. These datasets contained approximately 
1,316 participants including school students and preservice teachers. Most of them contained sample sizes of less 
than 100, and three contained 200 to 327 participants. These studies focused on almost all levels of grade 3-12 
education with 1,038 participants. The remaining studies were comprised of elementary and middle grade 
preservice teachers (n= 278). Most of the participants did not receive any incentive in terms of monetary or 
course credits but for students in grades 3-12, parental consent was noted. For preservice teacher education 
programs, the problem posing activities were conducted in their mathematics methods courses, while in grades 
3-12, the researchers utilized problem posing as an intervention strategy with arithmetic, algebra, number sense, 
and problem solving content. Many of the researchers incorporated problem posing using Brown and Walter’s 
(2005) What if not? into their classrooms. Some adapted the free, semi-structured, and structured problem posing 
situations strategies by Stoyanova and Ellerton (1996) and others used non-specific problem posing interventions 
in their studies. 

In terms of research design, the selected studies in this meta-analysis utilized quasi-experimental groups («=11) 
and pre-post contrasts ( n=2 ). Many of these researchers developed new instruments («=6) for data gathering 
while others (n= 7) used standardized (e.g., Iowa Test of Basic Skills-Problem Solving Subtest) or adapted 
instruments from previous research. In terms of research focus, most of the studies reported the effect of problem 
posing interventions on ability-based and attitude-beliefs learning outcomes. These included six studies that 
explored students’ levels and abilities to pose problems as well their interest toward mathematics. The remaining 
studies focused on the effect of problem posing interventions on students’ problem solving performance and 
mathematics achievement. 

5.2 Weighted Effect Size Indices 

Table 3 provides a general description illustrating the selected studies with comparisons between the weighted 
mean of effect sizes of the problem posing intervention based on student learning outcomes: knowledge-based 
(e.g., mathematics achievement), skill-based (e.g., problem solving performance), ability-based (e.g., levels of 
problem posed), and attitude-beliefs (e.g., interest and belief about/toward mathematics). 


Table 3. Mean study-weighted effect sizes 


Learning Outcomes 

#of weighted effect 
sizes 

Mean effect 
sizes 

Standard 

errors 

95% Cl 
(Lower) 

95% Cl 
(Upper) 

Knowledge-based 

4 

1.31 

.14 

1.04 

1.59 

Skill-based 

5 

.83 

.13 

.58 

1.08 

Ability-based 

7 

.77 

.10 

.57 

.98 

Attitude-beliefs 

6 

.76 

.09 

.59 

.93 


The results showed that the problem posing intervention had a positive and statistically significant impact on the 
student learning outcomes, specifically knowledge-based, skill-based, ability-based, and attitude-beliefs. The 
comparisons among the learning outcomes showed that problem posing had the largest positive impact on 
knowledge-based, mean Fledge’s g effect size of 1.31 with a 95% confidence interval of 1.04 to 1.59. 

5.3 Homogeneity of Effect Sizes and Publication Bias 

With Q statistic (chi-square) as a variance indicator, four studies under knowledge-based were heterogeneous 
wherein Q = 68.12 (df = 3, p < .001) indicating significant variability of one or more effects from the mean. In 
contrast, the impact of problem posing on student skill-based was large but Q = 7.06 {df= 4, p = .13) showing it 
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was not statistically significant indicating the studies were homogenous and could have come from the same 
distribution with a common mean. Also, the results of the Q statistic showed student learning outcomes, 
ability-based and attitude-beliefs were affected statistically (p < .001) by problem posing with Q = 24.65 (, df= 6) 
and Q = 71.78 (df= 5) respectively revealing significant heterogeneity in the mean effects. 

We sought to examine the funnel plots of each learning outcome by drawing smooth curves that connected the 
confidence limits of the effect sizes as displayed in Figure 1. The first funnel plot for knowledge-based indicates 
two studies, one at the bottom left and the other at the top right were outside the intervals of the mean effect. 
This indicated these studies were different from others which were also evidenced by the result of the Q statistic. 
Similarly, two to three effect size indices were not inside the confidence limits of ability-based and 
attitude-beliefs outcome, supportive of heterogeneity in the mean effects. In contrast, with respect to skill-based 
learning outcome, none were heterogeneous because all the studies were inside the confidence limits. In order to 
delve more deeply, we used ANOVA to examine the effects of the hypothesized moderator on these learning 
outcomes. 

5.4 Moderator Effects 

The results of the homogeneity analysis showed there might be some relationships between the learning 
outcomes (i.e., knowledge-based, skill-based, ability-based, attitude-beliefs) with the hypothesized moderators 
(i.e., educational setting, presentation type, research design, publication source). The discussion of moderator 
analysis will be presented according to each learning outcome based on the fixed effects model. 

5.4.1 Knowledge-Based 

Results in Table 4 showed that educational settings and publication sources had different effects statistically (p 
< .001) on student knowledge-based, indicating that the effect sizes were related to both moderators. In addition, 
the Q W ithi„ (Q = 4.95, p = .08) showed the effects of studies within grade 3-12 and teacher preparation groups 
could have come from the same distribution of effects. In contrast, the effect of publication source had 
heterogeneity variance overall within groups (Q= 42.93, p < .001) indicating some studies were statistically 
significantly different from each other. We excluded the moderator analysis for research design because these 
studies only used experimental and control groups, whereas the results of moderator analysis for presentation 
type were similar with the publication source. 
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Figure 1. Funnel plots of effect sizes according to knowledge-based, skill-based, attitude-beliefs, and 

ability-based (clockwise direction) 
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Table 4. Fixed effects model for knowledge-based within each moderator 

Moderator 

Group Sum of 

means weights 

SoS within group 

variation 

Qbetween 

(p-value) 

Qwithin 

(/j-valuc) 

Educational setting 

Grade 4-12 

.81 

33.89 

4.95 

106.48 

4.95 

Teacher preparation 

3.81 

32.12 

.00* 

A 

o 

o 

II 

© 

OO 

Publication source 






Published 

2.06 

49.13 

40.07 

58.22 

42.93 

Unpublished 

.64 

16.89 

2.86 

A 

o 

o 

A 

© 

o 

Note: *only one study involved. 





Table 5. Fixed effects model for skill-based within each moderator 



Moderator 

Group 

means 

Sum of 

weights 

SoS within group variation 

Qbetween 

(/;-value) 

Qwithin 

(/j-value) 

Educational setting 

Grade 4-12 

.80 

23.62 

.58 

21.29 

7.05 

Teacher preparation 

.86 

27.84 

6.47 

A 

© 

o 

r- 

p 

II 

3 

Publication source 






Published 

.79 

31.91 

7.19 

20.80 

7.33 

Unpublished 

.85 

21.11 

.13 

A 

© 

o 

II 

© 

ON 

Presentation type 

Structured pp. 

.65 

27.20 

2.85 

8.35 

2.85 

Non-specific pp. 

.44 

8.84 

.00* 

A 

© 

O = -42) 


Note: pp. = problem posing, *only one study involved. 


5.4.2 Skill-Based 

The results from the moderator analysis (see Table 5) presented the effectiveness of problem posing intervention 
on student skill-based learning outcomes were dependent on educational setting, publication source, and 
presentation type. The values of Q,demonstrated homogenous variances overall indicating that the effects of 
studies within each group emanates from the same distribution of effects. No moderator analysis was performed 
on research design because the studies utilized similar methodologies. 

5.4.3 Ability-Based 

All the hypothesized moderators had statistically (/? < .001) significant effects on student ability-based as shown 
in Table 6. Only presentation type showed homogeneity variance overall ( Q w uhin = 4.94, p = .42) indicating the 
studies within structured and non-specific problem posing might come from the same distribution of effects. 
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Table 6. Fixed effects model for ability-based within each moderator 


Moderator 

Group 

means 

Sum 

weights 

of SoS within group variation 

Qbetween 

(p-value) 

Qwithin 

(/> value) 

Educational setting 

Grade 4-12 

.69 

36.00 

18.99 

28.73 

23.81 

Teacher preparation 

.88 

36.16 

4.82 

A 

© 

o 

A 

© 

o 

Research design 

Independent group 

.77 

65.12 

24.62 

9.85 

24.62 

Matched-group 

.79 

7.05 

.00* 

A 

© 

o 

A 

© 

o 

Publication source 

Published 

.88 

65.61 

17.05 

13.44 

20.39 

Unpublished 

.35 

6.55 

3.35 

A 

© 

o 

A 

© 

o 

Presentation type 

Structured pp. 

.35 

17.46 

4.76 

40.16 

4.94 

Non-specific pp. 

1.27 

54.71 

.18 

A 

© 

o 

ip= -42) 


Note: pp.= problem posing, *only one study involved. 


5.4.4 Attitude-Beliefs 

Results of the fixed effect model for student attitude and beliefs within the hypothesized moderators is displayed 
in Table 7. The educational setting, research design, publication source, and presentation type had significant 
effects statistically (p < .001) on student attitude and beliefs toward mathematics. In addition, the ■,/„■„ in each 
category indicated heterogeneity variance overall showing some studies were significantly (p < .001) different 
from others. 


Table 7. Fixed effects model for attitude-beliefs within each moderator 


Moderator 

Group 

means 

Sum 

weights 

of SoS within group variation 

Qbetween 

(p-value) 

Qwithin 

(p-value) 

Educational setting 

Grade 4-12 

.54 

50.43 

11.69 

56.13 

79.26 

Teacher preparation 

1.28 

53.76 

67.56 

A 

© 

o 

A 

© 

o 

Research design 

Independent group 

.67 

81.17 

85.56 

37.33 

85.56 

Matched-group 

1.53 

22.99 

.00* 

A 

© 

o 

A 

o 

o 

Publication source 

Published 

.92 

75.17 

64.21 

35.47 

88.61 

Unpublished 

Presentation type 

.65 

27.62 

24.39 

(p<.001) 

A 

o 

o 
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Structured pp. .53 32.89 26.53 41.50 88.48 

Non-specific pp. .98 71.29 61.95 (/? < .001) (/? < .001) 

Note: pp. = problem posing, *only one study involved. 


6. Discussion 

This meta-analytic study advances current understanding of the effects of problem posing interventions on 
student outcomes in three valuable ways. First, it provides insights into the potential benefits of problem posing 
across the four coded outcomes of knowledge-based, ability-based, skill-based, and attitude and belief. Second, 
it provides valuable insights about the typical effect sizes one can expect for similar interventions in at least 
strong quasi-experimental studies. Third, the results show the power of being able to aggregate the studies based 
on Q statistics analyses and being able to control for publication bias in estimating the effects of problem posing 
on student learning. 

Initially the strong mean effects for ability-based, skill-based, and attitude-beliefs learning outcomes show that 
problem posing provides considerable benefits, accounting for nearly the same level of effect across these 
learning outcomes. The moderators for these three learning outcomes (1) educational settings (grade 3-12 versus 
teacher preparation); (2) publication sources (published versus unpublished); and (3) presentation types are all 
unequivocal. The three learning outcomes perform equally across the moderators with as much benefit for grades 
3-12 students as for preservice teachers, similar results for published versus unpublished sources, and across 
presentation types. Knowledge-based learning outcomes encompass a broad measure of mathematics 
achievement. The very large effect size for knowledge-based learning outcomes (1.31) is arguably the broadest 
measure of student success. Therefore, the case can be made that this learning outcome is possibly the most 
important of the three and exhibits the greatest effect across studies. 

These meta-analytic results are supportive of similar results in previous individual studies (e.g., Cai, 1998, 2003; 
Cai & Hwang, 2002; English, 1997, 1998; Lavy & Bershadsky, 2003; Moses et al., 1990; Silver et ah, 1996; 
Stoyanova, 1999). Further, the large positive effect sizes ranging from .76 to 1.31 provide reasonable and robust 
benchmarks when planning future studies, estimating sample sizes, and deciding on designs. These results are 
also important for classroom application and helping school leaders in decision making about implementation of 
a problem- posing program (cf. Lipsey & Wilson, 2001). School leaders should consider the four learning 
outcomes of knowledge-based, ability-based, skill-based, and attitude-beliefs, by reviewing student performance 
in these areas before they decide upon a problem-posing program. After due consideration of how students are 
performing in each area they can customize their program to the needs of students and have reasonable 
expectations for the effects they can expect from their efforts. This will make communication with teachers, 
parents and students more effective and provide a solid rationale for why they are adopting a problem-posing 
program. 

Preservice teachers, that are students in a teacher preparation program, were strongly influenced by 
problem-posing interventions across all learning outcomes. This result poses its own set of interesting and 
thought provoking questions. First, given the power of problem posing for preservice teachers, why has it not 
become more prominent in the best selling mathematics education methods textbooks? Second, given the weight 
of evidence for skill, ability, and attitude learning outcomes for preservice teachers, why is the topic of 
developing problem posing not more broadly implemented to help preservice teachers foster problem posing 
before they transition from education student to classroom teacher? Given only one study examining knowledge 
learning outcomes, it is difficult to claim robustness of the effect size estimate. However, this provides an 
opportunity for future research and potentially for dissertations and theses. It will be important to determine if 
other studies with different characteristics exhibit similar effects. Finally, other factors might be involved in 
causing the variations in the effect sizes for the learning outcomes; however, because of the lack of information 
reported in each study it is not possible to completely understand all factors. 

In general, it is promising to see that having people engaged in problem-posing activities can have broad 
potential benefits on students’ mathematics achievement, problem solving skills, levels of problems posed, and 
attitudes toward mathematics. This is the first quantitative synthesis on the topic, this study could not be 
compared with other meta-analyses for problem posing in the area of mathematics education. 

This meta-synthesis has several limitations. First, in terms of quality of the studies analyzed, there were few 
quantitative studies published on problem posing that included relevant data for calculation of effect sizes. Many 
studies did not provide detailed information about the problem-posing intervention that they used. The lack of 
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description makes replication difficult and aggregation more complex. In addition, some of the studies did not 
clearly present the results systematically. As a result, we had some difficulties analyzing the data provided from 
most of the studies. In fact, we excluded studies from which we were unable to extract the information after 
examining them. Secondly, only a small number of quantitative studies selected for this meta-analysis included 
randomization into groups. Many of the studies analyzed for this study implemented quasi-experimental designs 
with two-groups (experimental and control) and single-group (pre-post) designs. While using a pre-post design is 
better than posttest only but not as good as multiple observations, the studies provided sufficient detail for 
analyses but the drawback associated with their inclusion is that a one-group design tends to inflate effect sizes 
(Lipsey & Wilson, 2001; Shadish, Cook, & Campbell, 2002). The sample size is perhaps the most limiting aspect 
of this meta-analysis. The reason for the limited sample size was in part due to the fact that most studies were 
qualitative and did not report means and standard deviations. Even with the large proportion of qualitative 
studies, there were more than enough quantitative studies to provide robust estimates. The problem with the 
quantitative studies was that too few provided both adequate descriptions to code the learning outcomes and the 
means and standard deviations for the instruments used. To a lesser extent studies failed to provide adequate 
descriptions of the measures or failed to provide access to researcher-designed instruments. In order for 
meta-analyses to inform what we know about a topic, researchers must provide access to or include their custom 
designed instrumentation in the published work so it can be used in replication studies. If not, every study will 
likely become an N of 1. 

In conclusion, the results of this study are encouraging and suggest that the overall effects of problem posing 
interventions on teaching and learning of mathematics are positive and meaningful. Policy makers and educators 
should view this body of evidence when deciding on instructional strategies to implement in classrooms for 
improving students’: a) knowledge, b) problem solving skills, c) abilities to pose problems, and d) attitude 
toward mathematics at all levels. In addition, the results may stimulate future researchers to produce systematic 
studies on problem posing interventions. Still, many questions are unanswered related to the magnitude of effects 
of problem posing interventions on the learning of mathematics. Therefore, more extensive and effective 
research needs to be conducted in this area to enhance understandings for mathematics educators. 
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